{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 对活动数据进行分析\n",
    "(只取训练集和测试集中出现的样本)\n",
    "\n",
    "数据来源于 Kaggle 竞赛: Event Recommendation Engine Challenge, 根据\n",
    "    ①events they’ve responded to in the past\n",
    "    ②user demographic information\n",
    "    ③what events they’ve seen and clicked on in our app\n",
    "预测用户对某个活动是否感兴趣\n",
    "\n",
    "竞赛官网:\n",
    "https://www.kaggle.com/c/event-recommendation-engine-challenge/data\n",
    "\n",
    "\n",
    "活动描述信息在 events.csv 文件: 共 110 维特征  \n",
    "前 9 列: event_id, user_id, start_time, city, state, zip, country, lat, lng  \n",
    "event_id: 活动 ID   \n",
    "user_id: 创建活动的用户的 ID    \n",
    "city, state, zip, country: 举办活动地点位置的更多细节(如果有)  \n",
    "lat, lng: 浮点数(举办地点的经, 纬度坐标)  \n",
    "start_time: 字符串，ISO-8601 UTC 时间, 表示活动开始时间\n",
    "\n",
    "后 101 列为词频: count_1, count_2, ..., count_100, count_other  \n",
    "count_N: 活动描述出现第 N 个词的次数  \n",
    "count_other: 除了最常用的 100 个词之外的其余词出现的次数\n",
    "\n",
    "这里我们用 count_1, count_2, ..., count_100, count_other 属性做聚类, 即活动用这些关键词来描述, 可表示活动的类别"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 导入工具包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 数据量太大, pandas 不能一次将所有数据读入\n",
    "# 也可以用 pandas 一次读取部分数据, 可以参考: https://www.cnblogs.com/datablog/p/6127000.html\n",
    "import numpy as np\n",
    "import scipy.sparse as ss\n",
    "import scipy.io as sio\n",
    "\n",
    "# 保存数据\n",
    "import pickle\n",
    "\n",
    "# event 的特征需要编码\n",
    "from utils import FeatureEng\n",
    "from sklearn.preprocessing import normalize\n",
    "\n",
    "# 相似度/距离\n",
    "import scipy.spatial.distance as ssd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 统计活动数目"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of records: 3137972\n"
     ]
    }
   ],
   "source": [
    "# 读取数据, 并统计有多少不同的 events\n",
    "lines =0\n",
    "fin = open('events.csv', 'rb')\n",
    "fin.readline() # 跳过表头\n",
    "for line in fin:\n",
    "    cols = line.strip().split(','.encode(encoding='utf-8'))\n",
    "    lines += 1\n",
    "\n",
    "fin.close()\n",
    "print('Number of records: %d' %lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "活动数目太多(300w+), 训练+测试集的活动没这么多, 所以先去处理 train 和 test, 得到竞赛需要用到的活动和用户, 然后对在训练集和测试集中出现过的活动和用户建立新的 ID 索引. 先运行 user_event.ipynb, 得到活动列表文件: PE_eventIndex.pkl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取之前统计好的训练集和测试集中出现过的活动\n",
    "详见 1_Users_Events.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of events in train & test: 13418\n"
     ]
    }
   ],
   "source": [
    "# 读取训练集和测试集中出现过的活动集合\n",
    "eventIndex = pickle.load(open('PE_eventIndex.pkl', 'rb'))\n",
    "n_events = len(eventIndex)\n",
    "print('Number of events in train & test: %d' %n_events)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 处理 events.csv --> 特征编码、活动之间的相似度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "FE = FeatureEng()\n",
    "\n",
    "fin = open('events.csv', 'rb')\n",
    "fin.readline() # 跳过表头\n",
    "\n",
    "# start_time, city, state, zip, country, lat, lng\n",
    "eventPropMatrix = ss.dok_matrix((n_events, 7)) # 前 7 个特征矩阵\n",
    "\n",
    "# 词频特征矩阵\n",
    "eventContMatrix = ss.dok_matrix((n_events, 101))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for line in fin.readlines():\n",
    "    cols = line.strip().split(','.encode(encoding='utf-8'))\n",
    "    eventID = cols[0] # 第 1 列为活动 ID\n",
    "    \n",
    "    if eventIndex.__contains__(eventID): # 在训练集或测试集中出现(在活动索引的 key 中)\n",
    "        i = eventIndex[eventID]\n",
    "        \n",
    "        # event 的特征编码, 这里只是简单处理, 其实开始时间、地点等信息很重要\n",
    "        eventPropMatrix[i, 0] = FE.getJoinedYearMonth(cols[2]) # start_time 开始时间: 从 2010 年计月数\n",
    "        eventPropMatrix[i, 1] = FE.getFeatureHash(cols[3]) # city 城市\n",
    "        eventPropMatrix[i, 2] = FE.getFeatureHash(cols[4]) # state 州\n",
    "        eventPropMatrix[i, 3] = FE.getFeatureHash(cols[5]) # zip\n",
    "        eventPropMatrix[i, 4] = FE.getFeatureHash(cols[6]) # country 国家\n",
    "        eventPropMatrix[i, 5] = FE.getFloatValue(cols[7]) # lat 纬度\n",
    "        eventPropMatrix[i, 6] = FE.getFloatValue(cols[8]) # lng 经度\n",
    "        \n",
    "        # 词频\n",
    "        for j in range(9, 110):\n",
    "            eventContMatrix[i, j-9] = cols[j]\n",
    "            \n",
    "fin.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 用 L2 模归一化并保存\n",
    "eventPropMatrix = normalize(eventPropMatrix, norm='l2', axis=0, copy=False)\n",
    "sio.mmwrite('EV_eventPropMatrix', eventPropMatrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 词频, 可以考虑用这部分特征进行聚类, 得到活动的 genre\n",
    "# 用 L2 模归一化并保存\n",
    "eventContMatrix = normalize(eventContMatrix, norm='l2', axis=0, copy=False)\n",
    "sio.mmwrite('EV_eventContMatrix', eventContMatrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 基于两个矩阵计算 event pairs 之间的相似性\n",
    "eventPropSim = ss.dok_matrix((n_events, n_events))\n",
    "eventContSim = ss.dok_matrix((n_events, n_events))\n",
    "\n",
    "# 读取在训练集和测试集中出现过的活动对\n",
    "uniqueEventPairs = pickle.load(open('PE_uniqueEventPairs.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for e1, e2 in uniqueEventPairs:\n",
    "    i = e1\n",
    "    j = e2\n",
    "    \n",
    "    # 非词频特征, 采用 Pearson 相关系数作为相似度\n",
    "    if not eventPropSim.__contains__((i, j)):\n",
    "        epsim = ssd.correlation(eventPropMatrix.getrow(i).todense(), \n",
    "                               eventPropMatrix.getrow(j).todense())\n",
    "        eventPropSim[i, j] = epsim\n",
    "        eventPropSim[j, i] = epsim\n",
    "        \n",
    "    # 对词频特征, 采用余弦相似度, 也可以用直方图交/Jacard 相似度\n",
    "    if not eventContSim.__contains__((i, j)):\n",
    "        ecsim = ssd.cosine(eventContMatrix.getrow(i).todense(), \n",
    "                          eventContMatrix.getrow(j).todense())\n",
    "        eventContSim[i, j] = ecsim\n",
    "        eventContSim[j, i] = ecsim\n",
    "        \n",
    "sio.mmwrite('EV_eventPropSim', eventPropSim)\n",
    "sio.mmwrite('EV_eventContSim', eventContSim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[ 0.,  0.,  0., ...,  0.,  0.,  0.]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eventPropSim.getrow(0).todense()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
